Degenerate Bellman equation and its applications

Nonexistence of nonconstant solutions of some degenerate Bellman equations and applications to stochastic control

For a class of Bellman equations in bounded domains we prove that suband supersolutions whose growth at the boundary is suitably controlled must be constant. The ellipticity of the operator is assumed to degenerate at the boundary and a condition involving also the drift is further imposed. We apply this result to stochastic control problems, in particular to an exit problem and to the small di...

The Uncertainty Bellman Equation and Exploration

We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar uncertainty Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, ...

A Strong Comparison Result for the Bellman Equation Arising in Stochastic Exit Time Control Problems and Its Applications

Spurious Solutions to the Bellman Equation

Reinforcement learning algorithms often work by finding functions that satisfy the Bellman equation. This yields an optimal solution for prediction with Markov chains and for controlling a Markov decision process (MDP) with a finite number of states and actions. This approach is also frequently applied to Markov chains and MDPs with infinite states. We show that, in this case, the Bellman equat...

A Hybrid Bellman Equation for Bimodal Systems

In this paper we present a dynamic programming formulation of a hybrid optimal control problem for bimodal systems with regional dynamics. In particular, based on optimality-zone computations, a framework is presented in which the resulting hybrid Bellman equation guides the design of optimal control programs with, at most, N discrete transitions.